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This  documented  briefing  reports  on  the  “How  Will  the  e-Explosion  Affect  How 
We  Do  Research?”  Project.  In  particular,  it  documents  work  performed  during  Phase  I  of 
the  project,  including  descriptions  of  a  research  framework,  examples  of  e-based 
capabilities  associated  with  various  components  of  the  research  framework,  and  a  proof- 
of-concept  exercise  conducted  to  show  feasibility  and  effectiveness  of  an  e-based 
capability  called  “E-DEL+I.”  The  work  will  interest  those  involved  in  research  with 
participants  in  diverse  locations. 

This  research  results  from  RAND’s  continuing  program  of  self-sponsored 
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development  centers. 
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This  Independent  Research  and  Development  (IR&D)  project  is  entitled  “How 
Will  the  e-Explosion  Affect  How  We  Do  Research?”  For  the  purposes  of  this 
project,  “e-explosion”  means  the  proliferation  and  increased  accessibility  of 
computers  and  computer-based  capabilities,  such  as  the  Internet  and  software 
packages,  as  well  as  the  facilitation  this  type  of  automation  has  brought  to  many 
tasks. 

In  the  last  decade,  the  “e-explosion”  in  the  United  States  has  changed  nearly 
every  aspect  of  how  tasks  and  activities  in  the  business,  government,  and 
personal  worlds  are  conducted.  For  example,  customers  can  place  orders  for 
everything  from  clothing  to  groceries  on  the  Internet,  the  government  accepts 
proposals  electronically,  and  a  large  population  uses  e-mail  regularly  to 
communicate  with  friends  and  relatives.  With  the  emergence  of  network 
communications,  a  new  form  of  exchanging  ideas,  opinions,  and  results  has  also 
quietly  emerged  in  the  world  of  mathematical,  technological,  and  scientific 
research.  This  project  explores  how  research  might  be  affected  by  the 
e-explosion. 
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Outline 

►  Introduction 

•  Taxonomy  of  e-Based  Research  Tools 

•  Alternative  Implementation  Strategies 

•  E-DEL+I  Proof-of-Concept  Exercise 

•  Observations 

•  Appendix:  Detailed  Description  of 
Research  Framework 
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In  this  documented  briefing,  we  will  first  provide  an  introduction  to  the 
e-Explosion  Project,  including  its  two  phases.  That  discussion  will  be  followed 
by  a  description  of  the  taxonomy  work  accomplished  in  the  first  phase  of  the 
project.  Alternative  strategies  for  conducting  the  proof-of-concept  exercise, 
including  a  comparison  of  software  packages,  is  presented  next.  It  is  followed 
by  a  description  of  the  E-DEL+I  (Electronic  Decision  Enhancement  Leverager 
plus  Integrator)  proof-of-concept  exercise.  The  briefing  concludes  with 
observations  and  insights  drawn  from  the  proof-of-concept  exercise.  The 
appendix  contains  a  detailed  description  of  the  research  framework. 
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Purpose 

To  understand  how  the  “e-explosion”  will  affect  how 
research  is  performed  in  the  future 

•  Demonstrate  feasibility  of  e-based  research 
techniques  to  facilitate  research  with  participants 
in  diverse  locations 

-  Identify  candidate  techniques 

-  Conduct  proof-of-concept  experiments 

-  Evaluate  effectiveness 

•  Two  Phases 

-  Phase  I:  Research  framework  and  E+DEL+I  proof-of- 
concept  exercise 

-  Phase  II:  Taxonomy  of  e-based  capabilities  and  suite  of 
e-based  research  tools 
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The  purpose  of  this  project  is  to  understand  how  the  e-explosion  will  affect  how 
research  is  performed  in  the  future. 

The  participation  of  personnel  in  different  locations  can  often  enrich  research 
and  analysis  activities.  However,  in  science,  technology,  and  other  disciplines, 
research  and  analysis  have  largely  been  performed  at  single  sites  or  divided  into 
separate  tasks  if  multiple  sites  are  used.  Findings  of  the  multiple  tasks  are  then 
integrated  by  one  of  the  parties,  and  the  result  is  considered  collaborative 
research.  With  the  instantaneous  communication  capability  provided  by  the 
network,  ideas,  opinions,  and  results  of  ongoing  research  can  now  be 
transmitted  instantly  between  numerous  experts  all  over  the  world,  with 
unlimited  follow-up  on  ideas  expressed  in  the  exchanges.  Real-time 
collaborative  efforts  can  now  more  closely  resemble  scientists  truly  working 
side-by-side  to  accomplish  a  common  goal.  Thus,  e-communication  opens  up 
opportunities  to  perform  research  in  ways  different  from  the  traditional 
methods. 

While  research  using  e-based  capabilities  has  some  advantages,  some  problems 
might  also  make  this  approach  inappropriate  for  some  applications.  For 
example,  verification  of  information  might  be  more  difficult  and  work  in 
progress  might  be  disseminated  prematurely.  These  issues  should  be  considered 
before  deciding  to  use  an  e-based  capability.  The  appropriateness  of  using  an 
e-based  capability  versus  a  more  conventional  approach  should  be  decided  on  a 
case-by-case  basis. 
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E-based  approaches  to  research  are  not  new  to  RAND.  For  example,  in  1998 
RAND  researchers  conducted  a  successful  proof-of-concept  demonstration  of  a 
Web-based  science  and  technology  planning  technique  called  HyperForum. 

(See  R.  J.  Fempert  and  J.  Bonomo,  New  Methods  for  Robust  Science  and 
Technology  Planning,  Santa  Monica,  Calif.:  RAND,  DB-238-DARPA.) 

To  enhance  our  understanding  of  how  the  e-explosion  will  affect  how  we  do 
research,  this  project  will  demonstrate  the  feasibility  of  e-based  research 
techniques  by  showing  how  they  can  be  used  to  facilitate  research  with 
participants  in  diverse  locations.  We  will  identify  candidate  techniques, 
conduct  proof-of-concept  experiments,  and  evaluate  effectiveness. 

The  project  has  two  phases.  This  first  phase  encompasses  the  development  of  a 
research  framework  and  description  of  the  E-DEF+I  proof-of-concept  exercise. 
The  second  phase  will  include  a  taxonomy  of  e-based  research  capabilities  and 
description  of  a  suite  of  e-based  research  tools. 

This  documented  briefing  describes  work  accomplished  in  Phase  I.  It  includes  a 
description  of  the  research  framework.  Examples  of  e-based  capabilities  that 
can  be  associated  with  some  framework  elements  are  also  given  to  illustrate 
how  a  research  taxonomy  of  e-based  capabilities  can  be  formed.  The  E-DEL+I 
capability  is  described  in  the  report  as  a  consensus-building  technique.  The 
documented  briefing  concludes  with  observations  and  insights  from  the  proof- 
of-concept  exercise. 
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Project  Objectives 

•  Develop  a  research  taxonomy 

-  Based  on  research  framework 

-  Enables  organized  categorization  of  capabilities 

•  Existing,  evolving,  conceptual 

-  Enables  strategic  approach  to  enhancing  research 
capabilities 

•  Investigate  how  research  capabilities  can  be 
enhanced  by  exploiting  the  e-explosion 

-  Collaboration  from  diverse  locations 

-  Improve,  extend,  or  expand  existing  techniques 

-  Identify  new  analytic  capabilities 

-  Proof-of-concept  experiments 
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This  project  has  two  intertwined  objectives  that  will  be  accomplished  in  the  two 
phases  of  the  project.  This  documented  briefing  documents  work  on  both 
objectives  accomplished  during  the  first  phase.  The  first  objective  is  to  develop 
a  taxonomy  of  e-based  research  tools.  The  taxonomy  will  be  based  on  a 
research  framework.  The  framework  is  a  representation  of  the  interlocking 
steps  of  the  generic  research  process.  By  understanding  what  happens  at  each 
step,  we  can  focus  attention  on  what  e-based  capabilities  might  enhance 
performance. 

The  second  objective  of  the  project  is  to  investigate  potentially  beneficial 
e-capabilities.  In  particular,  we  are  interested  in  tools  that  will  allow  for 
collaborative  research  among  participants  in  diverse  locations.  We  are  also 
interested  in  established  research  techniques  that  might  be  improved,  extended, 
or  expanded  through  electronic  implementation.  Finally,  we  are  interested  in 
identifying  new  capabilities  made  possible  by  the  e-explosion.  When  we 
identify  a  capability  that  meets  one  or  more  of  these  criteria  and  existing 
technology  is  sufficient  to  demonstrate  the  capability,  we  will  design  and  carry 
out  proof-of-concept  experiments  to  confirm  the  usefulness  of  the  e-capability. 
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Outline 

•  Introduction 

►  Taxonomy  of  e-Based  Research  Tools 

•  Alternative  Implementation  Strategies 

•  E-DEL+I  Proof-of-Concept  Exercise 

•  Observations 

•  Appendix:  Detailed  Description  of 
Research  Framework 
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We  will  now  discuss  the  Phase  I  work  performed  on  the  taxonomy  of  e-based 
research  tools  task. 
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Taxonomy  of  e-Based  Tools 

•  Taxonomy  will  include  capabilities  that  are  of 
interest  to  the  Department  of  Defense,  the  Army, 
RAND,  the  greater  research  community,  and  the 
general  public 

-  Our  focus  will  be  on  intersection  of  RAND,  DoD,  and 
Army 

-  Alternative  paths  for  capabilities  of  generic  interest  will 
be  identified 

•  Taxonomy  will  enable  identification  of  areas  of 
focus 
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The  research  taxonomy  includes  capabilities  that  are  of  interest  to  the 
Department  of  Defense  (DoD),  the  Army,  RAND,  the  greater  research 
community,  and  the  general  public.  In  the  project,  we  will  focus  on  those 
capabilities  that  are  of  interest  to  the  DoD,  the  Army,  and  RAND.  Hence,  the 
types  of  research  we  will  focus  on  include  basic  scientific  research,  applied 
technical  research,  and  policy  analysis.  Those  capabilities  that  fall  outside  this 
intersection  will  remain  in  the  taxonomy,  but  no  proof-of-concept  experiments 
will  be  designed  to  demonstrate  their  usefulness. 

In  addition  to  identifying  potentially  useful  capabilities,  tools,  and  techniques, 
the  taxonomy  will  allow  for  identifying  areas  of  focus.  For  example,  if  our  goal 
were  to  use  e-techniques  for  hypotheses  formulation,  the  taxonomy  will  show 
what  capabilities  are  most  likely  to  contribute  to  that  goal.  In  this  manner,  the 
taxonomy  can  be  used  to  develop  strategies  for  exploiting  the  e-explosion. 
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Taxonomy  Is  Based  on  Research  Framework 

•  Framework  contains  interlocking  steps  that  make  up  the 
research  process 

•  Different  types  of  research  may  entail  different  activities  at 
each  step 

-  Framework  allows  for  the  many  types  of  research  and 
the  valuations  on  implementation  and  emphasis 

•  Framework  is  generic  and  reflects  the  research  process 

-  Research  at  RAND  and  Army  traverses  each  step 

•  Framework  gives  us  an  organized  manner  to  analyze  what 
goes  on  during  each  phase  of  research 

•  Framework  allows  us  to  identify  the  e-tools  that  might  help 
researchers  complete  each  phase  of  research 
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The  taxonomy  of  e-based  research  tools  is  based  on  a  research  framework.  The 
research  framework  contains  the  interlocking  steps  that  make  up  the  research 
process.  Different  types  of  research  may  entail  different  activities  at  each  step. 
The  framework  allows  for  the  many  types  of  research  and  the  variations  on 
implementation  and  emphasis. 

The  framework  we  have  developed  is  generic  and  reflects  a  research  process. 

In  this  representation,  research  traverses  each  interlocking  step.  In  particular, 
research  performed  at  RAND  and  research  performed  by  the  Army  traverses 
each  step,  although  different  organizations  or  groups  may  perform  each  step. 
This  research  framework  is  consistent  with  various  versions  of  the  scientific 
method. 

The  framework  allows  us  to  analyze  in  an  organized  way  events  during  each 
phase  of  the  research  process.  By  understanding  what  is  to  be  accomplished  at 
each  step,  we  are  then  prepared  to  identify  e-tools  that  might  enhance  the 
performance  of  each  step. 
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This  figure  depicts  the  research  framework  we  developed  for  the  taxonomy  of 
e-based  research  tools.  There  are  seven  basic  steps  to  the  research  process. 
These  steps  are  shown  above  in  the  middle  row  of  boxes.  A  research  effort 
begins  with  an  observation  that  generates  a  question  to  be  investigated.  This 
activity  is  followed  by  the  formulation  of  a  hypothesis.  The  researcher  then 
makes  a  prediction  about  the  hypothesis  and  proceeds  to  test  the  prediction. 
Analysis  of  the  test  results  follows.  That  analysis  is  followed  by  an 
interpretation  of  the  analysis  and,  finally,  the  researcher  draws  conclusions 
based  on  the  entire  research  process.  The  research  process  ends  with  one  or 
more  conclusions  regarding  the  question  generated  at  the  first  step. 

The  backward  arrows  at  the  bottom  of  the  boxes  indicate  the  iterative  nature  of 
a  typical  research  path.  There  may  not  be  a  one-time  clean  break  between  the 
steps.  Rather,  a  set  of  formal  or  informal  activities  may  help  transition  the 
research  to  the  subsequent  step. 

The  words  in  italics  at  the  top  of  the  figure  show  the  products  that  result  from 
the  steps  and  that  are  input  to  the  subsequent  step.  The  “make  observation  and 
generate  question”  step  results  in  a  question  that  is  input  to  the  “formulate 
hypothesis”  step.  The  “formulate  hypothesis”  step  results  in  a  hypothesis  input 
to  the  “make  prediction”  step.  The  “make  prediction”  step  results  in  a 
prediction  input  to  the  “test  prediction”  step.  The  “test  prediction”  step  results 
in  test  results  that  are  input  to  the  “analyze  test  results”  step.  The  “analyze  test 
results”  step  results  in  an  analysis  of  the  test  results  input  to  the  “interpret 
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analysis”  step.  The  “interpret  analysis”  step  results  in  an  interpretation  of  the 
analysis  input  to  the  “draw  conclusions”  step.  The  “draw  conclusions”  step 
results  in  one  or  more  conclusions  that  address  the  question  the  research 
considered. 

One  of  the  most  difficult  activities  to  place  was  the  “gather  and  analyze  data 
and  information”  activity.  As  the  figure  shows,  this  activity  is  actually 
performed  at  every  step.  Different  emphasis  and  methods  might  be  used  to 
perform  this  activity  at  the  different  steps. 

E-capabilities  can  enhance  or  facilitate  one  or  more  steps  of  the  research 
process.  By  associating  e-capabilities  with  the  steps  they  can  influence,  the 
research  framework  can  serve  as  a  structure  for  organizing  a  taxonomy  of 
e-capabilites. 

More  detailed  descriptions  of  each  of  the  steps  and  the  “gather  and  analyze  data 
and  information”  activity  are  presented  in  the  appendix:  “Detailed  Description 
of  the  Research  Framework.” 
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Examples  of  e-Based  Capabilities 
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This  figure  shows  examples  of  e-based  tools  currently  in  the  taxonomy.  For 
this  initial  phase  of  the  project,  we  are  concentrating  on  the  “make  prediction,” 
“test  prediction,”  “analyze  test  results,”  and  “gather  and  analyze  data  and 
information”  steps  of  the  research  framework. 

An  e- Acoustics  Library  is  an  example  of  a  tool  that  currently  does  not  exist  but 
is  conceivably  feasible  and  has  a  multitude  of  potential  uses.  Let  us  consider  a 
civilian  application  first.  Suppose  a  music  writer/researcher  wants  to  see  if  he 
can  copyright  a  tune  he  has  composed  or  a  researcher  wants  to  know  the 
particulars  of  a  specific  piece  of  music  but  can  only  hum  the  tune.  Currently, 
no  way  exists  for  either  person  to  input  a  series  of  notes  on  a  search  engine  and 
find  the  information  he  wants.  An  e-based  acoustics  library  would  be  suited  to 
inquiry  by  sound.  Incorporated  in  such  an  e-tool  would  be  a  classification 
method  to  quickly  identify  and  compare  different  sounds — a  “Dewey  decimal 
system”  for  sound.  Acoustics  experts  may  already  have  such  a  system,  but  it 
has  not  yet  been  adapted  for  Internet  use. 

Such  a  capability  would  present  a  host  of  issues,  such  as  how  different  a  new  set 
of  sounds  must  be  to  be  eligible  for  a  new  copyright  or  how  much  of  a 
copyrighted  piece  can  be  used  before  a  copyright  is  violated.  The  copyrighting 
or  trademarking  of  sounds  might  expand. 

This  concept  leads  to  a  possible  defense  application.  Could  monitoring  sound 
help  identify  what  weapons  or  vehicles  are  being  used  by  the  enemy?  Or 
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perhaps,  for  example,  if  Ford  Motor  Company  copyrighted  the  engine  sound  of 
its  Bronco,  it  could  then  license  the  use  of  that  sound  to  the  Army  for  use  in  its 
tanks.  The  science  of  monitoring  sound  for  intelligence  purposes  then  becomes 
more  complicated.  The  concept  of  countermeasures  with  respect  to  acoustics 
then  becomes  a  field  of  study. 

An  e-Holographic  Imaging  tool  could  employ  a  (yet-to-be-developed)  video 
camera  to  project  a  three-dimensional  scene  to  another  location.  Applications 
would  be  numerous,  with  scientists  in  other  locations  able  to  view  experiments 
virtually  in  person,  concert-goers  having  virtual  front-row  center  seats,  etc. 

A  third  e-technique  is  e-Logical  Accounting  in  which  experts  in  different 
specialties,  using  a  structural  approach,  can  contribute  toward  attaining  a 
specific  goal.  For  example,  Fermat’s  Last  Theorem  was  proved  by  a  single 
mathematician  who  devoted  seven  years  to  the  task.  An  oversight  was 
discovered  in  his  initial  proof,  but  it  was  finally  corrected  with  the  assistance  of 
another  mathematician.  If  the  mathematician  had  possessed  an  e-Logical 
Accounting  capability  whereby  mathematicians  could  review  his  lines  of 
reasoning  as  they  were  developed  and  offer  their  thoughts,  would  he  have  been 
able  to  address  the  oversight  before  his  initial  submission?  Or  could  the  proof 
have  been  accomplished  in  less  time? 

For  this  project,  we  focused  on  the  “Gather  and  analyze  data  and  information” 
portion  of  the  research  framework.  In  the  “Gather  and  analyze  data  and 
information”  activity,  the  researcher  is  selecting  and  acquiring  or  generating  the 
data  or  information  he  needs  to  perform  a  step  in  the  research  process. 
Consensus-building  among  a  panel  of  experts  is  one  method  for  generating  data. 
For  this  project,  we  focused  on  the  “Gather  and  analyze  data  and  information” 
portion  of  the  research  framework  and  developed  a  Delphi-inspired  electronic 
consensus-building  technique  called  the  Electronic  Decision  Enhancement 
Leverager  plus  Integrator  (E-DEL+I).  E-DEL+I  allows  for  four  rounds  of  a 
Delphi  exercise  to  be  conducted  in  the  course  of  a  typical  business  meeting  (two 
to  three  hours)  while  incorporating  both  written  and  oral  media  for  input  and 
maintaining  the  independence  and  anonymity  of  participants. 
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Definition  of  E-DEL+I  Process 

•  Electronic  real-time  enhancement  of  Delphi 
technique  for  establishing  consensus  positions 
among  experts  for  issues  encompassing  a  degree 
of  uncertainty 

•  Can  be  applied  to  policy  issues  that  require  in- 
depth  knowledge  and  forecasting  or  decision¬ 
making  based  on  that  knowledge 

•  We  used  the  E-DEL+I  process  to  assess  Army 
Basic  Research  Technologies  on  a  market 
breadth- Army  utility  framework 
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The  E-DEL+I  technique  is  an  electronic  real-time  enhancement  of  the  Delphi 
consensus-building  methodology.  This  tool  can  be  used  for  establishing 
consensus  positions  among  experts  for  issues  encompassing  a  degree  of 
uncertainty.  It  can  be  applied  to  policy  issues  that  require  in-depth  knowledge 
and  forecasting  or  decisionmaking  based  on  that  knowledge.  For  example, 
economic  and  technology  forecasts  are  often  based  on  Delphi  exercises. 

In  this  project,  we  used  the  E-DEL+I  technique  to  assess  Army  basic  research 
technologies  on  a  market  breadth-Army  utility  framework.  (A  description  of 
the  market  breadth-Army  utility  framework  if  given  on  p.  31  of  this 
documented  briefing.  See  Carolyn  Wong,  An  Analysis  of  Collaborative 
Research  Opportunities  for  the  Army,  Santa  Monica,  Calif.:  RAND,  MR-675-A, 
1998,  for  a  complete  description  of  the  market  breadth-Army  utility 
framework.) 
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E-DEL+I  Process 


Round  1  statistics  Round  2  statistics  and  Round  3  statistics  and 

and  justifications  minority  arguments  minority  defenses 


This  chart  shows  the  E-DEL+I  process.  The  E-DEL+I  process  has  four  rounds. 
In  the  first  round,  experts  complete  questionnaires  to  supply  assessments  based 
on  their  specialized  knowledge  and  background  material  provided.  The  exercise 
coordinator  computes  the  statistical  summary  of  the  first-round  inputs,  and  this 
summary — along  with  any  justifications  the  experts  provided  with  their 
assessments — is  the  feedback  the  exercise  coordinator  forwards  to  each 
participant  along  with  a  blank  questionnaire.  The  experts  review  the  feedback 
material  and  make  a  second  assessment,  this  time  supplying  arguments  for 
positions  in  the  minority  in  the  first  round. 

In  the  next  step,  however,  the  E-DEL+I  process  incorporates  a  real-time 
discussion  period  while  the  coordinator  generates  the  feedback  from  the  second 
round  of  inputs.  At  the  end  of  the  discussion  period,  a  statistical  summary  and 
minority  arguments  from  the  second  round  are  provided  along  with  a  third  blank 
questionnaire  for  the  third  round.  In  the  third  round,  participants  again  provide 
assessments  and  defenses  for  minority  positions  after  reviewing  the  feedback 
material  and  considering  what  was  presented  in  the  discussion  period.  A  second 
real-time  discussion  period  occurs  while  the  coordinator  compiles  the  statistical 
summary  and  minority  defenses  from  the  third  round.  The  summary  and 
minority  opinions  along  with  a  fourth  questionnaire  are  sent  to  the  participants  at 
the  end  of  the  discussion  period.  In  the  fourth  round,  the  experts  provide  their 
final  assessments  after  reviewing  and  considering  the  feedback  material  and 
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insights  gained  from  the  discussion  periods.  In  our  exercise,  the  statistical 
mode  will  be  used  as  the  consensus  position.  In  other  exercises,  the  mean  or 
median  may  better  reflect  the  consensus  position. 

In  our  exercise,  the  first  round  was  not  conducted  in  real  time.  This  was  a 
conscious  decision  to  allow  the  participants  an  opportunity  to  review  the 
background  material  adequately  and  familiarize  themselves  with  the 
questionnaire  format.  It  also  allowed  them  to  pose  questions  about  the  real-time 
sessions  to  follow.  All  subsequent  steps  were  conducted  in  real  time. 
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Outline 

•  Introduction 

•  Taxonomy  of  e-Based  Research  Tools 
►Alternative  Implementation  Strategies 

•  E-DEL+I  Proof-of-Concept  Exercise 

•  Observations 

•  Appendix:  Detailed  Description  of 
Research  Framework 
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We  will  now  discuss  the  alternative  implementation  strategies  considered  to 
conduct  the  proof-of-concept  exercise. 
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Comparison  of  Delphi  Implementations 


Time 

to 

complete 

Delphi 


On-site  meetings 


VTC 


Visits 


Mail 


Telephone 


Web  Delphi 
e-Delphi 


Password  chat  rooms 
Virtual  meeting 
E-DEL+I 


Number  of  sites/people  participating 


Existing  domain 
Higher  cost 

]  Exploratory  domain 
Lower  cost 
Existing  techniques 

Some  loss  of  anonymity 
Proof  of  concept 
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Before  describing  the  experiment  itself,  it  is  useful  to  know  how  Delphi 
implementations  have  been  carried  out  in  the  past — before  the  e-explosion. 
Delphi  exercises  have  typically  been  conducted  by  mail,  on-site  meetings,  in- 
person  visits,  or  telephone.  These  implementations  reside  in  the  generally 
higher-cost  existing  domain  shown  in  the  shaded  triangle. 

In  the  mail  implementation,  participants  are  mailed  questionnaires  and  asked  to 
complete  and  return  them  for  tabulation.  Feedback  material  and  a  new 
questionnaire  are  forwarded  to  each  participant  for  each  subsequent  round.  This 
process  can  take  months  to  accomplish  and  often  entails  reminder  postcards  and 
telephone  calls.  In  addition,  the  focus  of  the  experts  is  often  fragmented  by  the 
prolonged  exercise.  Also,  since  these  exercises  are  conducted  with  written 
input,  those  more  gifted  in  written  communications  can  have  an  advantage  over 
those  with  stronger  oral  communications  skills. 

On-site  meetings  involve  bringing  all  of  the  participants  together  to  one  site  at 
the  same  time.  In  this  setting,  the  exercise  can  be  conducted  in  writing  or 
orally.  However,  finding  a  time  when  all  the  participants  can  be  brought 
together  can  be  logistically  challenging  as  well  as  expensive. 

On-site  visits  and  telephone  interviews  are  less  common.  In  these  cases,  one  or 
more  coordinators  record  the  interview  responses.  The  accuracy  of  the 
recording  is  paramount  to  the  success  of  the  exercise.  Also,  discussions  cannot 
be  incorporated.  Because  one  or  more  coordinators  are  directly  involved  in 
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contacting  the  participants,  these  implementations  can  be  long  in  duration  or 
restricted  to  smaller  exercises. 

In  the  exploratory  domain  are  such  implementations  as  video-teleconferencing 
(VTC),  password  chat  rooms,  and  network  communications  packages,  such  as 
NetMeeting.  With  these  techniques,  some  loss  of  anonymity  is  more  likely 
because  the  participants  are  tagged  with  either  a  photo  identifier  in  VTC  or  a 
written  identifier  in  chat  rooms  and  network  communications  packages.  In 
these  cases,  steps  should  be  taken,  such  as  scrambling  of  identifiers  after  each 
round,  to  help  ensure  anonymity. 

Web  Delphi  and  e-Delphi  techniques  have  been  tried.  In  the  Web  technique, 
the  Internet  is  used  as  the  medium  and  the  exercise  and  the  material  are 
generally  posted  on  a  Web  site.  Participants  log  onto  the  Web  site  during  a 
specified  period  (e.g.,  a  week)  and  provide  inputs  at  times  compatible  with  their 
individual  schedules.  Subsequent  rounds  are  similarly  conducted.  This 
implementation  cannot  accommodate  real-time  discussions,  and  the  exercise  is 
conducted  with  written  communications,  possibly  giving  an  advantage  to  those 
with  strong  writing  skills.  Instant  Messaging  and  Internet  Chat  are  other  media 
that  are  growing  rapidly,  and  these  capabilities  may  also  play  roles  in  future 
Delphi-like  exercises. 

E-Delphi  is  a  technique  where  e-mail  is  used  to  conduct  the  exercise.  The 
participants  receive  exercise  material  via  e-mail  and  return  their  responses  using 
the  same  medium.  Discussions  are  by  e-mail,  slightly  delayed  from  real  time, 
and  the  exercise  is  conducted  with  written  communications  possibly  giving 
those  with  strong  writing  skills  an  advantage. 

Finally,  E-DEL+I  is  the  subject  of  our  proof-of-concept  exercise,  where  we  will 
incorporate  both  written  and  oral  communications  along  with  real-time  conduct 
of  the  exercise. 
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Relative  Cost  Comparison  of 
Delphi  Implementations 


Lower  Cost  Higher  Cost 
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This  figure  shows  the  relative  costs  of  conducting  a  Delphi  exercise  using 
various  implementations.  In  general,  the  electronically  based  implementations 
such  as  e-Delphi,  E-DEL+I,  Web  Delphi,  NetMeeting,  and  Password  Chat 
Rooms  are  less  costly  to  conduct  than  the  more  traditional  methods  such  as  on¬ 
site  meetings,  visits,  telephone,  and  mail.  Video  teleconferencing  (VTC)  costs 
can  vary  greatly  depending  on  access  to  facilities  providing  VTC  services. 
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Comparison  of  Network  Communications  Packages 
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We  examined  several  ways  to  conduct  the  proof-of-concept  exercise.  This  figure 
shows  a  comparison  of  the  various  alternatives  along  a  number  of  dimensions. 
Each  row  in  the  matrix  shows  a  different  alternative.  The  shaded  boxes  are 
elements  that  would  require  additional  costs  in  either  purchasing  or  leasing 
software  or  hardware  or  manpower  costs  that  would  be  in  addition  to  participant 
and  exercise  coordinator  costs.  Elements  in  italic  type  show  the  factors  that 
together  eliminated  the  option.  The  Aggregate  option  in  the  last  row  was 
selected  for  the  proof-of-concept  exercise. 

In  the  Aggregate  alternative,  Macs  and  PCs  could  be  accommodated.  Hence, 
participants  could  use  the  computer  equipment  they  already  had  in  their  offices. 
No  additional  hardware  or  software  would  be  required.  All-way  voice 
communications  would  be  established  through  a  conference  call  with  a  third- 
party  provider.  (For  this  exercise,  Genesys  Conferencing  provided  the 
conference  call  services  at  a  rate  of  $20  per  hour  per  participant.)  Participants 
would  use  e-mail  as  the  primary  method  for  sending  their  completed 
questionnaires  to  the  exercise  coordinator.  The  existing  inbox  capability 
established  by  the  RAND  computing  department  was  designated  as  a  backup  file 
transfer  mechanism  should  e-mail  become  unavailable  at  any  time  during  the 
real-time  exercise.  We  exploited  the  statistical  computation  capability  of  Excel 
to  compute  the  statistical  feedback.  Text  feedback  would  also  be  provided  via 
Excel  because  this  would  allow  the  participant  to  use  a  single  software  package 
to  complete  the  questionnaire  as  well  as  write  his  minority  arguments  and 
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defenses.  Compilation  of  the  participants’  questionnaires  would  be  a  manual 
procedure  performed  by  the  exercise  coordinator.  (In  a  second  exercise,  some 
Excel  software  macros  were  written  by  team  participant  David  Owen  to  allow 
for  semiautomated  compilation  of  questionnaire  responses.)  Finally,  the 
Aggregate  method  could  accommodate  the  hallmarks  of  Delphi  exercises — 
anonymity  and  independence  of  the  participants. 
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RAND  Network  Optimized  for  Throughput 
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This  figure  provides  an  overview  of  the  RAND  network.  One  of  the  reasons  for 
choosing  the  Aggregate  method  was  that  most  of  the  expert  participants  would 
be  RAND  employees  or  consultants  and  the  Aggregate  method  exploits  the 
capabilities  of  the  RAND  intranet.  The  RAND  intranet  is  optimized  to  provide 
maximum  throughput  between  points  regardless  of  the  protocol.  It  is  essentially 
one  or  two  T-l  lines,  each  of  which  provides  1.5  megabits  per  second  network 
throughput.  Our  preexercise  tests  showed  that  forwarding  questionnaires  as 
Excel  e-mail  attachments  met  the  real-time  constraints  of  the  proof-of-concept 
exercise. 
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Conceptual  NetMeeting  Network 
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By  comparison,  for  the  complementary  Microsoft  NetMeeting  package  to  work 
efficiently,  a  setup  similar  to  the  one  shown  on  this  figure  would  have  been 
needed  to  meet  the  constraints  of  the  proof-of-concept  exercise.  Likewise,  the 
other  existing  commercial  software  packages  could  not  be  efficiently 
accommodated  on  the  existing  RAND  network  without  incurring  substantial 
expenses  to  adapt  the  software. 
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We  will  now  discuss  the  E-DEL+I  proof-of-concept  exercise. 
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Purpose  of  Exercise 

•  Demonstrate  that  a  Delphi  exercise  can  be 
conducted  during  a  typical  business  meeting 

-  Participants  can  be  in  diverse  locations 

-  Nominal  time  span 

-  Discussions  without  sacrificing  anonymity  or 
independence 

-  Real-time  feedback 

•  Identify  challenges  in  multilocation  collaborative 
research 

•  Collect  data  for  Smart  Outsourcing  Project 
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Our  E-DEL+I  proof-of-concept  exercise  had  several  purposes.  One  was  to 
demonstrate  that  a  Delphi-like  exercise  can  be  conducted  during  a  typical 
business  meeting.  If  this  could  be  demonstrated,  then  a  phase  of  research  could 
be  accomplished  in  significantly  less  time  than  other  implementations  allow. 
For  our  exercise,  participants  would  be  in  diverse  locations.  In  fact,  each 
participant  took  part  in  the  exercise  from  his  or  her  own  office.  Offices  were  in 
Santa  Monica,  Arlington,  and  Pittsburgh.  (Computer  connections  were  actually 
made  through  the  RAND  Santa  Monica,  Washington,  and  Pittsburgh  offices  as 
well  as  from  the  University  of  Southern  California  in  Los  Angeles.  In  a  second 
exercise  using  this  implementation,  computer  connections  were  made  through 
the  three  RAND  offices  as  well  as  from  the  University  of  Southern  California 
and  from  Army  Materiel  Command  Headquarters  in  Arlington,  Virginia.)  We 
sought  to  demonstrate  that  a  phase  of  research  could  be  completed  in  a  nominal 
time  span — two  to  three  hours.  In  addition,  we  sought  to  show  that  real-time 
discussion  sessions  could  be  incorporated  without  sacrificing  the  anonymity  or 
independence  of  the  participants — the  hallmarks  of  a  Delphi  exercise. 
Furthermore,  we  wanted  to  show  that  it  is  possible  to  provide  real-time 
feedback  to  the  participants  to  enhance  their  assessment  abilities. 

Since  the  proof-of-concept  exercise  would  be  the  first  of  its  kind,  we  also 
wanted  to  identify  any  challenges  that  this  type  of  collaborative  research  effort 
might  have. 
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Finally,  we  collaborated  with  the  Arroyo  Center  Smart  Outsourcing  Project  and 
generated  data  needed  for  it,  completing  a  phase  of  research  for  that  project. 
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Design  of  E-DEL+I  Exercise 

•  Collaborated  with  Arroyo  Smart  Outsourcing  Project 

-  Use  E-DEL+I  to  place  Army  Basic  Research  Technologies  on  a 
market  breadth-Army  utility  framework 

•  Phase  1 — Preparatory  Session  and  Round  1 

-  Not  in  real  time 

-  By  e-mail 

-  Designed  to  encourage  participants  to  review  background  material 
and  familiarize  themselves  with  Excel  format 

•  Phase  2 — Rounds  2,  3,  and  4  with  discussions 

-  Real  time 

-  E-mail  for  instantaneous  transmission  of  files 

-  All-way  conference  call  for  voice  communications 
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The  purpose  of  the  Arroyo  Center  Smart  Outsourcing  Project  was  to  identify 
technologies  that  were  good  candidates  for  the  Army  to  use  for  various 
outsourcing  approaches  to  accomplish  research  goals.  It  included  a  task  to 
place  the  Army  Basic  Research  Technologies  on  a  market  breadth-Army  utility 
framework.  One  method  for  accomplishing  this  task  is  through  a  Delphi 
exercise.  We  collaborated  with  the  Smart  Outsourcing  Project  and  used  the 
E-DEL+I  method  to  accomplish  the  task.  Hence,  the  collaboration  would  allow 
the  Smart  Outsourcing  Project  to  complete  a  phase  of  research  at  the  same  time 
it  allowed  the  e-Explosion  Project  to  accomplish  its  proof-of-concept  exercise 
for  the  E-DEL+I  technique. 

We  divided  the  exercise  into  two  parts.  The  first  phase  consisted  of  a 
preparatory  session  and  the  first  round  of  the  exercise.  This  phase  was 
deliberately  not  conducted  in  real  time,  but  rather  by  e-mail  so  that  the 
participants  had  a  chance  to  ask  questions  about  the  E-DEL+I  process  and 
familiarize  themselves  with  the  questionnaire  format  and  e-mail  attachment  file 
transfer  process.  The  second  phase  consisted  of  Rounds  2,  3,  and  4  with 
discussion  periods.  The  second  phase  was  conducted  in  real  time  using  e-mail 
for  file  transmissions  and  an  all-way  conference  call  for  the  discussion  sessions. 
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Preparation  for  E-DEL+I  Exercise 

•  Selection  of  experts 

-  Experience/knowledge  regarding  Army  technologies, 
vision,  missions,  R&D  objectives  and  processes,  and 
operational  philosophy 

-  Experience/knowledge  of  business  R&D  practices  and 
commercial  motivational  concepts 

-  Availability  to  participate  in  all  four  E-DEL+I  rounds 

•  E-DEL+I  preparation  session 

-  Definitions  of  framework  and  Army  basic  technologies 

-  Explanations  of  process  and  assessment  form 

•  Independence  and  anonymity 

•  Coordinator  feedback  for  the  multiple  rounds  and  schedule 
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One  of  our  first  tasks  was  to  select  the  experts  who  would  participate  in  the 
proof-of-concept  exercise.  We  worked  with  the  Smart  Outsourcing  Project 
team  to  identify  suitable  participants.  The  criteria  for  selection  included 
experience  and  knowledge  of  Army  technologies,  vision,  missions,  research  and 
development  objectives  and  processes,  and  Army  operational  philosophy.  In 
addition,  the  participants  were  required  to  have  some  degree  of  experience  or 
knowledge  of  business  research  and  development  practices  and  commercial 
motivational  concepts.  Finally,  the  participant  had  to  be  available  to  participate 
in  both  phases  of  the  exercise. 

We  offered  a  preparation  session  to  all  participants.  During  this  session,  we 
reviewed  the  definitions  of  the  market  breadth-Army  utility  framework  and  the 
Army  Basic  Research  Technologies.  In  addition,  we  provided  oral  explanations 
of  the  E-DEL+I  process  and  reviewed  the  assessment  form  (i.e.,  the 
questionnaire).  We  emphasized  the  importance  of  the  participants’ 
maintenance  of  independence  and  anonymity  in  providing  their  assessments  and 
explained  the  role  of  the  coordinator.  In  addition,  we  informed  the  participants 
that  an  assistant  would  be  available  should  anyone  have  e-mail  problems  or 
problems  with  conference  call  procedures.  (Janie  Young  of  RAND  served  as 
the  assistant  during  the  exercise.)  All  of  this  information  was  provided  in 
writing  by  e-mail  prior  to  the  preparation  session.  Hence,  attending  the 
preparation  session  was  optional. 
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E-DEL+I  Assessment 

•  Expert  panel  consisted  of  13  RAND  staff  and  consultants 

•  Four  rounds  of  E-DEL+I  to  place  12  Army  Basic  Research 
Technologies  in  a  predefined  market  breadth-Army  utility 
framework 

•  Two  monitored  discussion  sessions  with  ground  rules 

•  Speakers  cannot  identify  themselves 

•  Speaker  cannot  directly  specify  assessment 

•  Round  1  conducted  electronically  by  e-mail 

•  Rounds  2,  3,  4  conducted  in  real  time  by  network  and  phone 
connections 

-  Panel  located  in  RAND  Santa  Monica,  Washington,  and 
Pittsburgh  offices 
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The  E-DEL+I  assessment  consisted  of  four  rounds  of  expert  assessments  to 
place  twelve  Army  Basic  Research  Technologies  on  a  predefined  market 
breadth-Army  utility  framework.  The  expert  panel  of  participants  consisted  of 
13  RAND  staff  and  consultants.  Round  1  was  conducted  electronically,  where 
participants  returned  their  completed  first-round  questionnaires  to  the  exercise 
coordinator  by  e-mail.  Rounds  2,  3,  and  4  were  conducted  in  real  time  on 
January  19,  2001.  Expert  panel  members  participated  from  their  Santa  Monica, 
Washington,  and  Pittsburgh  RAND  offices.  (One  consultant  established 
computer  connections  through  the  University  of  Southern  California.) 

Two  discussion  sessions  were  included  as  part  of  the  exercise.  The  first 
discussion  session  occurred  between  the  second  and  third  rounds.  The  second 
discussion  session  occurred  between  the  third  and  fourth  rounds.  We  specified 
two  ground  rules  for  the  monitored  discussion  sessions.  The  first  rule  was  that 
the  speaker  could  not  identify  himself  either  by  name  or  by  job  title.  The 
second  rule  was  that  the  speaker  could  not  directly  specify  his  assessment.  We 
imposed  these  rules  to  ensure  the  anonymity  and  independence  aspects  of  the 
exercise.  (Bruce  Held  of  RAND  served  as  the  discussion  monitor.)  No 
violations  of  the  ground  rules  occurred. 
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Army  Basic  Research  Technologies 

Synthesized  from  the  Army  Science  and  Technology  Master  Plan  and 
Descriptive  Summaries  of  Research,  Development,  Test,  and 
Evaluation  Army  Appropriation  Budget  Activities  1,  2,  and  3 

•  Propulsion  and  Vehicular  Mobility 

•  Materials  and  Mechanics 

•  Ballistics 

•  Sensors 

•  Air  Mobility 

•  Applied  Physics 

•  Information  and  Communications 

•  Medical 

•  Engineering  and  Construction 

•  Environment  and  Soil 

•  Human  Engineering 

•  Personnel  Equipment,  Performance,  and  Training 
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The  Army  Basic  Research  Technologies  that  the  expert  panel  placed  on  the 
market  breadth-Army  utility  framework  are  shown  on  this  figure. 
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Framework  and  Domains 


□  Lead — Army  unique 

ED  Join — -joint  with  other  services, 
DoD,  or  government  agencies 

[|  Initiate — collaborate  with 
industry;  moderate  to  high 
Army  utility 

I  Participate — collaborate 
with  industry;  moderate 
to  low  Army  utility 

9  Monitor — leave  to  industry, 
Army  monitors 


Army  unique  Joint  Generic 

Market  Breadth 
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High 


Army  Utility 


Low 


This  figure  shows  the  market  breadth-Army  utility  framework.  Market  breadth 
means  the  amount  of  appeal  the  technology  has  to  the  commercial  sector  and 
hence  how  interested  other  sectors  might  be  in  performing  basic  research  in  the 
technology.  Army  utility  is  the  amount  of  potential  contribution  the  technology 
has  in  helping  the  Army  accomplish  its  military  mission. 

The  framework  is  composed  of  five  domains:  Lead,  Join,  Initiate,  Participate, 
and  Monitor.  The  E-DEL+I  exercise  asked  each  member  of  the  expert  panel  to 
independently  and  anonymously  place  each  of  the  twelve  Army  Basic  Research 
Technologies  in  the  domain  that,  in  that  expert’s  judgment,  best  reflects  the 
technology’s  market  breadth  and  Army  utility. 

A  “Lead”  placement  indicates  that  the  technology  has  limited  appeal  outside  of 
the  Army  and  hence  the  Army  must  perform  most  of  the  basic  research  in- 
house. 

A  “Join”  placement  indicates  that  the  technology  is  of  interest  to  other  military 
or  government  agencies  and  an  effective  approach  is  for  the  Army  to  perform 
basic  research  in  the  technology  as  joint  projects  with  one  or  more  of  the  other 
interested  government  entities. 

An  “Initiate”  placement  indicates  that  the  technology  has  moderate  to  high 
Army  utility  and  also  appeals  to  industry.  The  Army  should  collaborate  with 
industry  but  might  want  to  initiate  the  effort  to  help  ensure  that  its  military- 
specific  needs  are  met. 
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A  “Participate”  placement  indicates  that  the  technology  has  of  moderate  to  low 
Army  utility  and  also  appeals  to  industry.  The  Army  should  collaborate  with 
industry  to  accomplish  basic  research  goals  in  these  technologies. 

A  “Monitor”  placement  indicates  that  the  technology  has  moderate  to  low  Army 
utility  and  has  high  industry  appeal.  The  Army  can  rely  on  industry  to  perform 
basic  research  in  these  technologies,  supplying  little  or  no  Army  resources. 
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Example  of  Preparation  Material 

Basic  Research  Technology  Descriptions 

Propulsion  and  Vehicular  Mobility:  Propulsion  research  is  a  joint  effort  sponsored  by  the  National 
Aeronautics  and  Space  Administration  and  the  Department  of  Defense.  It  is  focused  on  turboshaft 
engine-specific  technology  and  mechanical  power  transmission  technology,  as  applicable  to  rotorcraft 
and  tracked  and  wheeled  vehicles.  Analysis,  code  generation,  experiments,  and  evaluations  are  conducted 
to  improve  engine  and  drivetrain  components  and  investigate  advanced  materials.  Component-level 
investigations  include  compressors,  combustors,  turbines,  injectors,  pistons,  cylinder  liners,  piston  rings, 
gears,  seals,  bearings,  shafts,  and  controls.  The  goal  of  the  activity  is  increased  performance  of  small  air- 
breathing  engines  and  power  trains  that  will  support  improvements  in  system  mobility,  reliability,  and 
survivability  and  ultimately  serve  to  reduce  the  logistics  cost  burden  on  future  concepts,  including  the 
Future  Combat  Systems  (FCS)  program.  Vehicle  dynamics  and  simulation  and  advanced  track  and 
suspension  concepts  research  focus  on  power  density,  performance,  and  thermal  efficiency  for  advanced 
adiabatic  diesel  engines,  transient  heat  transfer,  high-temperature  materials,  and  thermodynamics. 
Optimized  parameterization  procedures  are  directly  applicable  to  the  FCS. 

Materials  and  Mechanics:  The  goal  of  materials  research  is  to  establish  the  science  base  allowing  the 
creation  and  production  of  advanced  materials  that  will  provide  higher  performance,  lower  cost, 
improved  reliability,  and  environmental  compatibility  for  Army-unique  applications.  Emphasis  is  on 
understanding  the  fundamental  aspects  of  chemistry  and  microstructure  that  influence  the  performance 
and  failure  mechanisms  of  ceramics,  advanced  polymer  composites,  advanced  metals,  and 
multifunctional  materials.  These  advanced  materials  will  enable  lethality  and  survivability  technologies 
for  the  FCS. 
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This  figure  shows  an  example  of  the  background  material  that  was  provided  to 
all  participants.  Participants  received  the  material  as  e-mail  attachments.  In 
addition,  the  material  was  orally  reviewed  during  the  optional  preparation 
session. 
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Rating  Scale 


1  n  Lead — Technology  has  limited  industry  appeal.  Army  performs  research  in-house. 

2  □  Join — Technology  of  interest  to  other  military  or  government  agency.  Army  performs  research 

jointly  with  other  agencies. 

3  □  Initiate — Technology  of  moderate  to  high  Army  utility  appeals  to  industry.  Army  collaborates  with 

industry  in  R&D. 

4  □  Participate — Technology  of  moderate  or  low  Army  utility  appeals  to  industry.  Army  collaborates 

with  industry  in  R&D. 

5  |  Monitor — Technology  of  moderate  to  low  Army  utility  has  high  industry  appeal.  R&D  performed  by 

industry  with  little  or  no  Army  resources. 

Discrete  scale  was  used 
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This  figure  shows  the  predetermined  rating  scale  the  participants  used  to  assess 
placement  of  the  technologies.  For  each  Army  Basic  Research  Technology,  the 
participant  would  designate  the  integer  from  one  to  five  that  he  felt  best  reflects 
that  technology’s  market  breadth  and  Army  utility. 
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Round  2  Assessment  Form 


Statistical  Feedback  from 
Round  1  Responses 

FY2001  Army  Basic  Technology 

Domain 

Outline  reasoning  if  Round  2  response  does 
not  match  Round  1  mode  or  if  there  is  more 

than  one  Round  1  mode 

Mode(s) 

Mean 

Median 

Propulsion  &  Vehicular  Mobility 

3 

2.615385 

3 

Materials  &  Mechanics 

3 

2.769231 

3 

Ballistics 

2 

1.692308 

2 

Air  Mobility 

2,  3 

2.615385 

3 

Applied  Physics 

5 

4 

4 

Information  &  Communications 

3 

3 

3 

Medical 

3 

3.538462 

3 

Engineering  &  Construction 

3 

2.923077 

3 

Sensors 

2 

2.692308 

3 

Environment  &  Soil 

4 

2.923077 

3 

Human  Engineering 

4 

3 

3 

Personnel  Performance,  Training,  &  Equipment 

4 

2.923077 

3 
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This  figure  shows  the  Round  2  Assessment  form.  The  E-DEL+I  exercise 
assessment  forms  were  developed  in  Excel.  Excel  was  chosen  because  all 
participants  already  had  this  software  package  installed  on  their  computers; 
Excel  has  a  well-known  interface;  Excel  is  very  easy  to  use  as  a  prototyping 
tool;  and  Excel  could  accommodate  the  needs  of  the  proof-of-concept  exercise 
with  minimal  software  development  costs.  In  addition,  computation  of 
statistical  feedback  would  be  facilitated  by  Excel’s  built-in  statistical  functions. 
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Overall  Consensus  Formation 


At  least  seven  on  panel  agreed 

At  least  nine  on  panel  agreed 

Number  of 
Technologies 

Percentage 

Number  of 
Technologies 

Percentage 

Round  1 

3 

25 

1 

8 

Round  2 

6 

50 

3 

25 

Round  3 

11 

92 

5 

42 

Round  4 

11 

92 

7 

58 
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This  figure  shows  the  results  of  each  round  of  the  exercise.  The  second  column 
shows  the  number  of  technologies  where  at  least  seven  expert  panel  members 
agreed  on  placement  in  the  framework  for  each  round.  The  associated 
percentages  are  shown  in  the  third  column.  The  fourth  column  shows  the 
number  of  technologies  where  at  least  nine  expert  panel  members  agreed  on 
placement  in  the  framework  for  each  round.  The  associated  percentages  are 
shown  in  the  fifth  column.  Using  the  results  shown  in  either  the  second  column 
or  the  fourth  column,  this  figure  shows  that  the  feedback  and  discussion 
sessions  helped  establish  consensus  positions  for  more  technologies  in  each 
subsequent  round.  Using  the  results  shown  in  the  second  column,  consensus 
was  reached  on  1 1  of  the  12  technologies  in  Round  4.  Using  the  results  shown 
in  the  fourth  column,  consensus  was  reached  on  7  of  the  12  technologies  in  the 
final  round. 
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How  the  Mode  Changed 


Mode 

Technology 

Round  1 

Round  2 

Round  3 

Round  4 

Propulsion  and  Vehicular  Mobility 

3 

3 

3 

Materials  and  Mechanics 

3 

3 

3 

Ballistics 

2 

2 

2 

Air  Mobility 

2,  3 

2 

2 

Applied  Physics 

5 

5 

5 

Information  and  Communications 

3 

3 

3 

Medical 

3 

3 

4 

Engineering  and  Construction 

3 

3 

3 

Sensors 

2 

3 

3 

Environment  and  Soil 

4 

4 

4 

Human  Engineering 

4 

4 

4 

Personnel  Performance,  Training,  and  Equipment 

4 

4 

2 
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For  this  exercise,  the  mode  was  used  as  the  consensus  position  statistic.  Since 
we  are  placing  technologies  on  a  framework,  where  most  experts  placed  the 
technology  (i.e.,  in  the  Lead,  Join,  Initiate,  Participate,  or  Monitor  domains)  is 
the  best  indication  of  a  consensus  position  among  them.  Because  the  statistical 
mode  of  each  round  is  mathematically  defined  as  the  category  with  the  greatest 
number  of  placements  by  the  experts,  this  statistic  is  the  most  appropriate 
measure  of  consensus  for  this  exercise.  This  figure  shows  how  the  mode 
changed  for  each  technology  for  each  round.  The  lighter-shaded  boxes  show 
how  either  dual  modes  changed  to  single  modes  or  single  modes  changed  to 
dual  modes  in  the  subsequent  round.  The  darker-shaded  boxes  show  single 
modes  changing  to  other  single  modes  in  the  subsequent  round.  The  unshaded 
boxes  show  constant  modes  from  one  round  to  the  subsequent  round. 
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Round  1  to  Round  2  Position  Changes 


Round  1 


Round  2 


Lead 

Join 

Initiate 

Participate 

Monitor 

Lead 

5 

4 

Join 

1 

9 

1 

Initiate 

3 

9 

1 

Participate 

5 

1 

Monitor 

1 

1 

•  40  out  of  132  (30  percent)  responses  changed  between  Round  1 
and  Round  2 

•  Four  participants  did  not  change  any  positions  between  Round  1 
and  Round  2 
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This  figure  shows  how  the  participants  changed  their  positions  between  Round 
1  and  Round  2.  For  example,  one  expert  changed  his  placement  of  one 
technology  from  Join  to  Lead  from  Round  1  to  Round  2.  Likewise,  three 
Round  1  Initiate  votes  were  changed  to  Join  in  Round  2.  In  total,  40  out  of  132 
(30  percent)  of  the  assessments  were  changed  between  the  first  two  rounds. 
Four  participants  did  not  change  any  of  their  assessments  for  any  technology 
between  the  first  and  second  round. 
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Round  2  to  Round  3  Position  Changes 


Round  3 


Round  2 


Lead 

Join 

Initiate 

Participate 

Monitor 

Lead 

6 

1 

1 

Join 

6 

Initiate 

3 

3 

Participate 

1 

5 

1 

Monitor 

2 

4 

•33  out  of  132  (25  percent)  responses  changed  between  Round  2  and 
Round  3 

•  Two  participants  did  not  change  any  positions  between  Round  2  and 
Round  3 
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Thirty-three  out  of  132  assessments  (25  percent)  were  changed  between  Round 
2  and  Round  3.  Two  participants  did  not  change  any  assessments  between  the 
second  and  third  round. 
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Round  3  to  Round  4  Position  Changes 


Round  3 


Round  4 


Lead 

Join 

Initiate 

Participate 

Monitor 

Lead 

1 

Join 

3 

Initiate 

3 

1 

Participate 

2 

3 

Monitor 

1 

•  14  out  of  132  (1 1  percent)  responses  changed  between  Round  3  and 
Round  4 

•  Three  participants  did  not  change  any  positions  between  Round  3 
and  Round  4 
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Fourteen  out  of  132  assessments  (11  percent)  were  changed  between  Round  3 
and  Round  4.  Three  participants  did  not  change  any  positions  between  Round  3 
and  Round  4. 
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Strong  First-Round  Consensus  with  Further 

Buildup 

Information  and  Communications  Consensus-Building 


— ♦ — Round  1 
Round  2 
—  ▲ —  Round  3 
Round  4 


Lead  Join  Initiate  Participate  Monitor 

Domain 


Most  information  and  communications  basic  research  can  be  pursued 

via  Army  Initiate. 
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Seven  different  patterns  of  consensus-building  were  detected  in  this  exercise. 
For  this  analysis,  we  used  the  criterion  that  a  consensus  is  achieved  for  a 
technology  when  at  least  seven  members  of  the  expert  panel  placed  the 
technology  in  the  same  domain  in  the  framework.  We  use  the  term  “strong 
consensus”  to  mean  that  more  than  seven  members  of  the  expert  panel  placed 
the  technology  in  the  same  domain  in  the  framework.  In  addition,  the  term  “no 
consensus”  means  that  less  than  seven  members  of  the  expert  panel  placed  the 
technology  in  the  same  domain  in  the  framework.  The  patterns  range  from  a 
strong  consensus  in  the  first  round  and  holding  or  building  for  the  remaining 
rounds  to  no  consensus  in  the  early  rounds  and  a  strong  consensus  by  the  final 
round.  This  figure  and  the  next  six  figures  describe  the  consensus-building 
patterns. 

This  figure  shows  the  consensus-building  process  for  the  Information  and 
Communications  technology.  In  this  case,  the  panel  members  indicated  that 
most  information  and  communications  research  can  be  pursued  via  an  “Army 
Initiate”  approach.  That  is,  the  Army  initiates  basic  research  efforts  in  this 
technology.  (See  Carolyn  Wong,  An  Analysis  of  Collaborative  Research 
Opportunities  for  the  Army,  Santa  Monica,  Calif.:  RAND,  MR-675-A,  for  more 
details  on  the  Lead,  Join,  Initiate,  Participate,  and  Monitor  aspects  of  the  market 
breadth-Army  utility  framework.)  In  this  case,  most  panel  members  agreed  on 
the  Initiate  placement  in  the  first  round.  Subsequent  rounds  only  increased  this 
initial  consensus. 
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Multiple  First-Round  Positions  Consolidating 
to  Dominant  Position  at  Second  Round 

Materials  and  Mechanics  Consensus-Building 


— * — Round  1 
— ■ —  Round  2 
. .  .4. . .  Round  3 
Round  4 


Lead  Join  Initiate  Participate  Monitor 

Domain 


Most  materials  and  mechanics  research  can  be  pursued  via  Army  Initiate. 
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For  the  Materials  and  Mechanics  technology,  the  first-round  assessments  were 
spread  among  all  of  the  placement  categories.  The  feedback  material  provided 
between  the  first  two  rounds  helped  achieve  a  consensus  in  the  second  round. 
The  Round  2  feedback  and  first  discussion  session  raised  the  consensus  level  in 
the  third  round,  and  the  consensus  position  topped  out  without  changing  again 
in  the  fourth  round.  In  this  case,  the  consensus  position  was  not  established 
until  the  second  round  and  it  remained  consistent,  building  up  to  the  fourth 
round  near-unanimous  assessment.  At  the  end  of  the  exercise,  all  but  one 
expert  agreed  that  most  Materials  and  Mechanics  research  can  be  pursued  with 
an  Army  Initiate  approach. 
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First  Discussion/Comments  Solidified  Weak 
Round  2  Consensus 

Sensors  Consensus-Building 


— ♦ — Round  1 
Round  2 
-  a  -  Round  3 
Round  4 


Lead  Join  Initiate  Participate  Monitor 

Domain 


•  Lead  supported  by  some  sensors  that  are  critical  and  only  of  Army  interest 

•  Join  supported  by  sensors  critical  to  Army  vision  and  because  commercial  sector  alone 
cannot  meet  Army  FCS  needs 

Most  sensors  work  can  be  Initiate,  but  some  thrusts  better  suited  for  Join  or  Lead. 
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In  the  Sensors  technology,  the  expert  assessments  were  scattered  among  the 
Lead,  Join,  Initiate,  and  Participate  positions  for  the  first  two  rounds.  This 
indicates  that  the  feedback  material  provided  between  Rounds  1  and  2  did  not 
convince  many  to  change  their  initial  assessments.  A  significant  change 
occurred  between  the  second  and  third  rounds.  The  third-round  assessments 
show  a  clear  “Initiate”  consensus  position.  We  can  infer  that  the  written 
feedback  as  well  as  the  first  discussion  session  imparted  convincing  arguments 
that  spurred  nearly  a  quarter  of  the  expert  panel  to  change  their  assessment  to 
the  Initiate  position.  In  this  case,  no  consensus  was  established  until  the  third 
round,  and  the  fourth  round  brought  only  a  minor  one- vote  increase  to  the 
consensus  position.  The  feedback  and  initial  discussion  session  were  key  to 
establishing  a  consensus  position  for  the  Sensors  technology.  (Note  that  this 
pattern  shows  the  dangers  of  conducting  truncated  two-round  Delphi  exercises 
as  mail  implementations  often  do  to  decrease  the  duration  of  the  exercise.) 

In  this  case,  the  justifications,  minority  arguments,  and  defenses  contained 
material  useful  to  the  Smart  Outsourcing  Project.  The  experts  supporting  the 
Lead  position  consistently  argued  that  some  sensors  are  critical  and  are  only  of 
interest  to  the  Army.  The  Join  position  supporters  cited  that  sensors  are  critical 
to  the  Army  vision  and  that  the  commercial  sector  alone  cannot  meet  Army 
Future  Combat  System  (FCS)  needs.  They  advocate  a  position  that  the  most 
effective  Army  approach  to  accomplish  basic  research  in  sensors  are  joint 
efforts  by  the  Army  and  other  military  services. 
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First  Discussion/Comments  Scattered  Positions;  Second 
Discussion/Comments  Consolidated  Positions 


Human  Engineering  Consensus-Building 


— ♦ —  Round  1 
Round  2 
Round  3 
Round  4 


•  Join  cite  combat  man/machine  situations  unique  to  military 

•  Initiate  cites  commercial  sector  alone  not  suited  for  man/machi  ne  realities 

Much  basic  research  in  human  engineering  is  suited  for  Participate,  but  some 
collaborations  need  Army  leadership  and  some  research  is  best  siited  to  joint  efforts. 
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The  exercise  results  show  yet  a  different  consensus-building  pattern  for  the 
Human  Engineering  technology.  In  this  case,  the  first  two  rounds  did  not 
produce  consensus  positions.  The  material  in  the  feedback  to  Round  3  and  first 
discussion  session  between  the  second  and  third  round  split  the  panel  into  two 
camps — one  favoring  the  Join  position  and  the  other  favoring  the  Participate 
position.  Feedback  from  the  third  round  and  the  second  discussion  session 
convinced  eight  participants  to  vote  for  the  Participate  position.  Hence,  in  this 
case,  the  strong  consensus  was  not  reached  until  the  fourth  round. 

Once  again,  the  justifications,  minority  arguments,  and  defenses  provided 
information  useful  to  the  Smart  Outsourcing  Project.  Those  supporting  the  Join 
position  stated  that  combat  man/machine  situations  are  unique  to  the  military 
and  hence  that  joint  efforts  with  other  military  services  constitute  the  best 
approach  for  the  Army  to  accomplish  basic  research  goals  in  human 
engineering.  The  Initiate  proponent  presented  a  similar  argument  stating  that 
the  commercial  sector  alone  is  not  suited  to  basic  research  for  man/machine 
realities.  This  expert’s  Initiate  position  indicates  that  he  believes  the 
commercial  sector  can,  however,  play  a  significant  role. 
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Second  Discussion/Comments  Dispersed  Weak 
Round  3  Consensus 

Applied  Physics 


— # —  Round  1 
— ■ —  Round  2 
. .  -A- . .  Round  3 
Round  4 


Lead  Join  Initiate  Participate  Monitor 

Domain 


•  Participate  supported  by  a  lot  of  Army-industry  overlap,  but  enough  Army  uniqueness 
for  Army  to  define  needs  and  not  necessarily  supply  a  lot  of  Army  funding 

•  Monitor  supported  by  majority  of  work  in  areas  mentioned  being  done  by  industry 

Most  applied  physics  R&D  suitable  for  Participate  or  Monitor,  but  collaborative  work 
in  some  areas  should  be  initiated  by  Army. 
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The  Applied  Physics  technology  area  shows  an  exercise  pattern  in  which  the 
consensus  actually  became  weaker  in  the  final  round.  A  predominance  for  the 
Monitor  position  was  indicated,  but  no  consensus  position  was  established  in 
the  first  round.  However,  the  second  and  third  rounds  are  marked  by  nearly 
identical  results  and  a  Monitor  consensus  position.  Hence  the  feedback  from 
the  first  round  convinced  some  experts  to  change  their  position  to  Monitor.  The 
feedback  from  the  third  round  and  the  initial  discussion  period  did  not  change 
the  consensus  strength.  However,  the  feedback  from  the  third  round  and  the 
second  discussion  session  broke  the  consensus,  and  the  fourth  round  resulted  in 
no  final  consensus  position — only  a  split  between  the  Participate  and  Monitor 
positions.  Supporters  of  the  Participate  position  stated  that  much  overlap  can  be 
found  between  Army  and  industry  in  this  area,  but  uniqueness  in  Army  basic 
research  goals  in  applied  physics  warrants  the  Army  to  define  its  needs  and  not 
necessarily  supply  a  lot  of  funding  for  basic  research  toward  those  needs.  The 
Monitor  group  cited  that  the  majority  of  work  going  on  in  the  applied  physics 
area  is  already  being  done  by  industry  and,  hence,  that  the  Army  can  rely  on  the 
Monitor  approach. 
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Second  Discussion/Comments  Consolidated 
Scattered  Positions 


Personnel  Performance,  Training,  and  Equipment  Consensus  Formation 


4 —  Round  1 
a —  Round  2 

4 _ Round  3 

Round  4 


•  Initiate  cite  nature  of  combat  stress  requires  Army  leadership,  high  Army  utility,  and  great 
importance  in  industry 

Much  basic  research  in  personnel  performance,  training,  and  equipment  is  suited  for 
joint  efforts,  but  some  is  suited  for  Initiate. 
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For  the  Personnel  Performance,  Training,  and  Equipment  technology  area,  the 
exercise  results  were  scattered  for  the  first  three  rounds.  The  Join  consensus 
position  was  reached  after  the  participants  received  feedback  from  Round  3  and 
took  part  in  the  second  discussion  session.  In  the  fourth  round,  all  experts 
supported  either  the  Join  or  Initiate  positions.  The  Initiate  supporters  pointed 
out  that  the  nature  of  combat  stress  requires  Army  leadership.  In  addition,  there 
is  high  Army  utility,  but  this  area  is  also  of  great  importance  to  industry.  From 
these  comments  and  the  exercise  results,  the  Smart  Outsourcing  team  was  able 
to  conclude  that  much  basic  research  in  Personnel  Performance,  Training,  and 
Equipment  technology  is  suited  for  joint  efforts  with  other  military  services,  but 
some  is  suited  for  the  Initiate  approach  with  industry. 
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Positions  Stabilized  by  Round  3 


Environment  and  Soil  Consensus-Building 


— ♦ —  Round  1 
— ■ —  Round  2 
— * —  Round  3 
Round  4 


Lead  Join  Initiate  Participate  Monitor 

Domain 


•  Lead  and  Join  cite  importance  to  weapon  design 


Much  of  environment  and  soil  basic  research  is  suited  for  Participate,  but  some 
collaborations  need  to  be  led  by  the  Army. 
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This  figure  shows  the  consensus-building  pattern  for  the  Environment  and  Soil 
technology  area.  In  this  case,  a  Participate  consensus  position  was  established 
in  the  third  round.  Hence,  feedback  from  the  second  round  and  the  initial 
discussion  period  consolidated  the  expert  panel  viewpoints.  The  fourth -round 
returns  were  identical  to  the  third-round  returns,  indicating  that  the  feedback 
from  the  third  round  and  the  second  discussion  session  did  not  change  anyone’s 
assessment  in  this  technology  area. 

The  Lead  and  Join  dissenters  consistently  cited  the  importance  of  this  area  to 
weapon  design  as  their  argument  for  the  Lead  or  Join  positions. 
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Placement  of  Technologies 


High 


Army  Utility 


Low 


Lead 
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•Air  Mobility 
•Personnel 
Performance, 
Training,  & 
Equipment 


Join  . 


Army  Unique 


Propulsion  &  Vehicular  Mobility 
•Materials  &  Mechanics 
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•Medical 
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•Human  Engineering 
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Joint 

Market  Breadth 


Generic 
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The  previous  seven  figures  showed  the  consensus-building  patterns  for  selected 
technologies.  This  figure  shows  the  final  framework  placements  for  all  the 
technologies  using  the  criterion  that  a  consensus  was  achieved  if  at  least  seven 
members  of  the  expert  panel  placed  the  technology  in  the  same  domain  in  the 
framework.  Note  that  the  Applied  Physics  technology  is  shown  in  both  the 
Participate  and  Monitor  domains,  indicating  the  lack  of  consensus  and  bimodal 
predominances  of  the  final  results  for  this  technology  area. 

As  an  indication  of  sensitivity  to  the  criterion  used  for  concluding  that  a 
consensus  was  reached,  only  seven  of  the  technologies  would  be  shown  on  this 
figure  as  having  achieved  consensus  positions  if  we  had  used  the  criterion  that 
consensus  requires  at  least  nine  experts  placing  the  technology  in  a  particular 
domain.  At  the  extreme,  no  consensus  positions  would  be  shown  if  for  each 
technology  we  required  all  13  panel  members  to  agree  before  declaring  a 
consensus. 
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Outline 

•  Introduction 

•  Taxonomy  of  e-Based  Research  Tools 

•  Alternative  Implementation  Strategies 

•  E-DEL+I  Proof-of-Concept  Exercise 
►Observations 

•  Appendix:  Detailed  Description  of 
Research  Framework 
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We  will  now  discuss  observations  and  insights  gained  from  the  E-DEL+I  proof- 
of-concept  exercise. 
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Exercise  Results 

•  E-DEL+I  proof-of-concept  exercise  showed  Delphi 
process  can  be  conducted  in  typical  business  meeting  with 
participants  in  diverse  locations 

-  Discussions  useful  for  clarifications  and  conveying  information 

-  Independence  and  anonymity  preserved  through  indirect  contact 

-  Balanced  approach  successful 

-  Newcomers  and  acquaintances  worked  well  together 

-  “Miles”  unnoticed 

•  Successfully  generated  data  for  Smart  Outsourcing  Project 
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The  E-DEL+I  proof-of-concept  exercise  took  two  hours  and  45  minutes  to 
complete,  illustrating  that  this  approach  can  complete  a  phase  of  research 
during  a  typical  business  meeting  with  participants  in  diverse  locations. 
Furthermore,  the  discussion  sessions  were  useful  both  for  clarifications  and  for 
conveying  information.  Some  positions  were  changed  as  a  result  of  the 
discussion  sessions.  With  our  two  basic  ground  rules,  we  were  able  to  preserve 
the  independence  and  anonymity  hallmarks  throughout  the  exercise  because  the 
participants  only  had  indirect  contact  via  the  conference  telephone  call.  The 
balanced  approach  in  which  participants  with  strong  writing  skills  could  use  a 
written  medium  to  explain  their  positions  and  those  with  strong  speaking  skills 
could  use  the  real-time  oral  discussion  sessions  to  explain  their  positions 
worked  well.  Informal  postexercise  interviews  with  the  participants  indicated 
that  all  felt  they  had  adequate  opportunities  and  comfortable  means  to  convey 
their  thoughts,  although  some  did  express  a  preference  for  writing,  whereas 
others  said  they  just  jotted  notes  in  their  written  responses,  preferring  to  address 
the  group  during  the  discussion  sessions. 

The  expert  panel  consisted  of  colleagues  who  had  worked  closely  together  as 
well  as  newcomers  unacquainted  with  any  other  participant.  However,  the 
newcomers  felt  that  their  input  was  welcome  and  did  not  detect  any  bias  against 
their  participation.  That  we  appeared  to  achieve  a  level  playing  field  for  all 
participants  is  evidenced  by  postexercise  comments,  such  as  “I  didn’t  know 
who  was  talking,  so  I  had  to  assume  that  he  knew  what  he  was  talking  about.” 
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In  addition,  the  “miles”  between  the  physical  locations  appeared  to  go 
unnoticed  during  the  exercise,  demonstrating  that  with  the  appropriate  tools, 
research  can  be  effectively  and  successfully  accomplished  with  participants  in 
diverse  locations. 

Finally,  the  exercise  successfully  completed  a  research  task  for  the  Smart 
Outsourcing  Project,  generating  the  information  that  project  was  seeking. 

We  did  not  conduct  a  parallel  exercise  to  see  how  the  consensus-building 
patterns  or  results  of  the  proof-of-concept  exercise  might  compare  with  a 
conventionally  conducted  Delphi.  We  believe,  however,  that  the  balanced 
approach  and  discussion  sessions,  elements  not  available  in  conventional 
exercises,  helped  the  experts  make  well-informed  judgments.  In  addition,  the 
only  expense  we  incurred  that  would  not  have  been  incurred  in  a  conventional 
exercise  was  the  cost  of  the  all-way  conference  call  (less  than  $900),  while  a 
conventionally  conducted  exercise  would  have  incurred  many  higher 
expenditures  that  the  E-DEL+I  exercise  did  not  incur.  For  example,  the  costs  of 
bringing  the  13  researchers  to  a  single  physical  location  for  a  conventional  on¬ 
site  exercise  would  have  been  at  least  $6,000  for  travel  expenses  alone. 
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Feedback 


•  Responses  required  14  to  29  minutes 

•  More  exercise  time 

-  Pressure  to  finish 

•  Discussions  useful 

-  If  not  convinced,  clarified  my  thinking 

-  “I  didn't  know  that”  conveyance  of  useful  information 

•  Anonymity,  independence  preserved 

-  Did  not  know  who  was  talking 

-  Realized  it  should  not  matter  who  was  talking 

•  Would  do  it  again 

-  Sense  that  phase  of  research  could  be  completed  quickly 
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The  exercise  participants  were  informed  of  the  agenda  but  were  not  given 
specific  durations  to  complete  each  round  or  discussion  session.  Participants 
took  between  14  and  29  minutes  to  complete  each  assessment  questionnaire 
(rating  the  technologies  plus  supplying  any  supporting  arguments).  Even 
though  the  participants  did  not  receive  any  time  limits  and  were  told  to  take  as 
much  time  as  they  needed,  postexercise  interviews  indicated  that  several 
participants  felt  that  they  were  under  pressure  to  complete  the  questionnaires 
quickly.  In  retrospect,  some  of  this  feeling  might  have  been  generated  because 
the  discussion  sessions  were  scheduled  to  begin  as  soon  as  a  quorum  of 
participants  had  completed  the  questionnaires  and  had  something  to  say 
(determined  by  the  discussion  monitor).  The  participants  completed  the 
questionnaires  with  the  conference  call  on  hold,  so  they  were  not  aware  when 
the  discussions  would  start  but  would  simply  join  in  after  completing  their 
responses.  Nevertheless,  a  few  participants  felt  that  if  they  did  not  complete  the 
questionnaire  quickly,  they  would  miss  out  on  the  discussion  session.  This 
shows  that  some  participants  viewed  the  discussion  sessions  as  valuable. 

Further  indications  that  the  participants  valued  the  discussion  sessions  are 
postexercise  interview  comments  stating  that  the  discussions  helped  clarify 
thinking  even  if  the  participants  were  not  persuaded  to  change  their  positions. 
Conveying  information  appeared  to  be  a  major  role  of  the  discussion  sessions 
because  in  postexercise  interviews  some  participants  said  they  were  exposed  to 
different  views  of  technology  areas  that  they  themselves  knew  something  about. 
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Anonymity  and  independence  appeared  to  have  been  preserved.  Postexercise 
interviews  showed  that  the  participants  were  not  always  sure  who  was  talking, 
and  some  indicated  that  they  finally  realized  that  it  should  not  matter  who  was 
talking. 

The  final  question  in  the  postexercise  interview  was  whether  the  participant 
would  participate  in  another  such  exercise.  All  participants  indicated  that  they 
would. 


53 


Unanticipated  Challenges 

•  Not  everyone  thoroughly  reads  e-mail 

-  Three  unaware  of  Phase  1  resulting  in  one 
dropout 

•  Technical  annoyances 

-  File  transfer  incompatibilities 

-  Preset  telephone  settings 

-  Conference  call  etiquette 

•  Formal  preparation  session  desirable 
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Although  the  feedback  from  the  exercise  participants  was  generally  positive  and 
the  exercise  did  accomplish  its  purposes,  not  everything  happened  as  expected. 

Our  first  surprise  was  that  not  everyone  thoroughly  reads  e-mail.  Fourteen 
experts  initially  agreed  to  participate  in  the  exercise.  After  the  deadline  for 
returning  the  first-round  responses,  three  responses  were  missing.  We  contacted 
the  participants  who  had  not  responded.  All  three  acknowledged  receiving  the  e- 
mail  announcing  the  start  of  the  exercise.  However,  they  had  not  read  it 
thoroughly  enough  to  realize  that  they  were  supposed  to  complete  the  first-round 
questionnaire.  Two  of  these  individuals  then  furnished  their  missing  responses, 
and  one  decided  not  to  participate  (resulting  in  13  participants). 

There  were  also  some  technical  difficulties.  The  first  was  that  we  found  file 
transfer  incompatibilities  among  the  different  platforms  being  used  by  the 
participants.  Fortunately,  dropping  the  file  into  the  exercise  coordinator’s  inbox 
worked  successfully  for  the  e-mail  attachments  that  could  not  be  read.  (A  variant 
of  this  solution,  such  as  anonymous  drop  boxes,  might  be  required  if  participants 
unable  to  return  responses  by  e-mail  did  not  have  access  to  RAND  internal  drop 
boxes.)  Second,  we  found  that  the  telephones  at  RAND  have  preset  settings  that 
had  to  be  changed  for  the  conference  call  to  take  place.  Fortunately,  this 
problem  was  discovered  prior  to  the  exercise,  and  we  were  able  to  send 
telephone  instructions  to  all  the  participants  in  advance.  Finally,  none  of  the 
participants  had  very  much  experience  with  13- way  conference  calls  and 
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conference  call  etiquette  involving  such  issues  as  background  noise  (e.g., 
tapping  with  a  pencil,  putting  down  a  coffee  cup)  had  to  be  relayed  to  ensure 
that  the  telephones  did  not  cut  speakers  off  in  attempts  to  pick  up  the 
background  noise. 

Although  all  participants  were  sent  background  material  and  exercise 
participation  instructions,  most  preferred  a  formal  preparation  session  in  which 
the  exercise  coordinator  explained  each  item  rather  than  going  through  the 
material  on  their  own. 
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Pros  and  Cons  of  E-DEL+I 

•  Allows  for  collaborative  research  from  diverse  locations 

•  A  phase  of  research  can  be  completed  at  minimal  cost  and  time 

•  Retains  anonymity  and  independence  hallmarks  of  Delphi 

•  Variations,  such  as  discussion  periods,  can  be  included 

•  Balanced  approach  allows  those  with  strong  verbal  or  strong 
writing  skills  to  use  preferred  medium 

•  Preparation  prior  to  real-time  session  essential 

-  Coordinator 

-  Participants 

•  E-DEL+I  requires  all  participants  to  be  available  during  a  set 
time  period 

-  Time  zone  differences  can  dictate  changes  to  normal  schedule 
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The  E-DEL+I  proof-of-concept  exercise  shows  that  it  can  be  used  for 
collaborative  research  with  participants  in  diverse  locations.  Using  this 
technique,  a  phase  of  research  can  be  completed  at  minimal  cost  and  in  a 
minimal  amount  of  time.  Our  implementation  of  the  technique  allows  for 
retention  of  the  anonymity  and  independence  hallmarks  of  Delphi  exercises. 

We  demonstrated  that  discussion  sessions  can  be  valuable  input  mechanisms. 
Finally,  the  balanced  approach  uniquely  incorporated  into  the  technique  allows 
participants  with  strong  verbal  or  writing  skills  to  participate  using  their 
preferred  medium. 

On  the  negative  side,  preparation  prior  to  the  real-time  session  is  essential.  The 
coordinator  must  design  and  send  out  the  questionnaires  and  background 
material  well  before  the  real-time  session.  In  addition,  the  participants  must 
review  the  material  prior  to  the  real-time  session.  While  such  preparations  are 
no  different  from  those  that  participants  in  any  type  of  business  meeting  might 
be  expected  to  make,  the  coordinator  had  no  opportunity  to  refine  or  revise 
questionnaire  processing  procedure  because  of  the  real-time  nature  of  the 
exercise.  In  a  conventional  exercise,  the  coordinator  could  revise  processing 
procedures  after  receiving  all  of  the  questionnaires  or  could  even  reprocess 
them.  The  participants  must  also  be  available  during  a  specific  period  on  a 
specific  date.  Time  differences  can  cause  inconveniences.  For  example,  by 
majority  preference,  the  participants  in  our  proof-of-concept  exercise  who  were 
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physically  located  on  the  East  Coast  voluntarily  shifted  their  noon  lunch  hour  to 
participate.  (The  proof-of-concept  exercise  took  place  from  noon  to  2:45  p.m., 
Eastern  Standard  Time.) 
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Insights 

•  E-DEL+I  is  one  of  a  suite  of  e-based 
research  tools 

•  Accomplishing  research  using  e-tools  is 
possible  and  often  desirable  but  comes  at  a 
cost 

•  Exercise  experience  suggests  potentially 
useful  capabilities/tools/studies 

-  Some  are  research,  others  general  application 
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Our  experience  with  E-DEL+I  yielded  some  insights.  The  first  is  that  E-DEL+I 
can  be  viewed  as  one  of  a  suite  of  e-based  research  tools,  much  as  Word  is  one 
of  the  tools  in  the  Microsoft  Office  Suite.  Second,  distributed  data  generation 
for  research  is  possible  and  has  some  distinct  advantages,  but  a  cost  is  also 
involved.  Finally,  the  proof-of-concept  exercise  suggests  some  potentially 
useful  capabilities,  tools,  and  studies,  some  of  them  research-oriented  and 
others  with  a  more  general  application.  The  following  figures  will  elaborate  on 
each  of  these  insights. 
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E-DEL+I  Is  One  of  a  Suite  of 
Research  Tools 

•  Similar  to  Microsoft  Office  Suite 

•  Alternative  paths  toward  reality 

-  Develop  within  RAND  for  RAND  and  Army  or  client  use  only 

-  Develop  within  RAND  available  with  or  without  fee  to  public 

-  License  implementation,  outside  developer,  share  benefits 

•  RAND  experience  enables  us  to  specify  requirements, 
desirable  features,  options 

-  For  example,  statistics  generation,  prose  formatting,  directed 
tutorial  versus  questionnaire 

-  Might  be  some  intersection  with  survey  requirements 
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E-DEL+I  can  be  viewed  as  one  capability  in  a  suite  of  research  tools  much  as 
Word  or  Excel  is  a  capability  in  the  Microsoft  Office  Suite.  The  taxonomy  of 
e-based  capabilities  can  be  used  to  identify  other  candidate  capabilities  to 
include  in  such  a  suite. 

Should  this  path  be  followed,  several  alternatives  are  available  to  make  such  a 
suite  a  reality.  One  is  to  develop  the  suite  of  tools  within  RAND  for  RAND  and 
Army  or  client  use  only.  Alternatively,  the  suite  can  be  developed  within 
RAND  but  made  available  to  the  public  with  or  without  user  fees.  Third,  the 
suite  can  be  licensed  for  outside  development,  where  RAND  would  share  the 
benefits  with  the  developer. 

Whatever  path  is  chosen,  RAND  should  remain  a  participant  because  its  unique 
collective  experience  in  research  and  analysis  enables  RAND  to  specify 
requirements,  desirable  features,  and  options  that  should  be  included  in  a  suite 
of  e-capabilities  for  research.  For  example,  statistics  generation,  prose 
formatting,  directed  tutorials,  or  survey  requirements  might  be  desirable 
attributes  for  research  tools. 
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Proposed  Research  Suite  Study 

•  Identify  spectrum  of  realistic  alternatives 

-  Might  be  different  for  different  categories  of  e-capabilities 

•  For  example,  if  it  has  Army  applicability  versus  being  a  generic  tool 

•  Determine  potential  RAND/client  benefits  for  each 
alternative 

-  For  example,  would  providing  a  free  E-DEL+I  capability  be 
consistent  with  RAND’s  mission? 

•  Determine  resource  requirements  for  each  alternative 

-  Long-term  and  near-term  money,  time,  personnel,  etc. 

•  Make  recommendations 

-  Base  recommendation  on  pilot  development  of  E-DEL+I  capability 
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A  recommended  follow-on  study  should  include  such  tasks  as  identifying  the 
spectrum  of  realistic  alternatives.  There  might  be  different  categories  of  e- 
based  capabilities.  For  example,  some  may  be  applicable  only  to  the  military, 
whereas  others  have  more  generic  applications.  Once  the  alternatives  are 
identified,  a  determination  should  be  made  of  the  potential  for  each  alternative 
in  terms  of  benefits  to  RAND  or  its  clients.  For  example,  would  providing  a 
free  E-DEL+I  capability  be  consistent  with  RAND’s  mission?  Once  the 
potential  benefits  are  known,  resource  requirements  for  the  most  promising 
alternatives  need  to  be  determined.  For  example,  what  are  the  long-term  and 
near-term,  time,  resource,  and  dollar  requirements?  Finally,  the  study  should 
make  recommendations  on  how  to  pursue  a  pilot  development  of  the  E-DEL+I 
capability. 
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E-Based  Research  Requirements 
on  Management 

•  Budget  for  e-based  capability 

•  Plan  for  consequences  of  adding  capability 

-  Learning  time 

-  Might  take  more  than  one  try  for  success 

•  Additional  or  revised  policies  may  be 
appropriate 

-  For  example,  file  sharing,  security,  proprietary, 
drafts 
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Some  management  moves  are  required  to  use  e-tools  successfully  to  conduct 
research  with  participants  in  diverse  locations.  First,  such  collaboration  must  be 
included  in  the  effort’s  budget.  Second,  because  using  e-tools  to  conduct 
research  with  participants  in  diverse  locations  is  a  new  way  of  completing 
research,  management  should  plan  for  possible  consequences  of  adding  such  a 
capability.  For  example,  researchers  might  need  a  learning  period  to  get  used  to 
working  in  this  manner.  As  such,  more  than  one  try  might  be  required  for 
success.  Finally,  additional  or  revised  corporate  policies  may  be  required.  For 
example,  policies  on  file  sharing,  computer  security,  and  sharing  of  draft 
documents  may  have  to  be  modified. 

Of  course,  collaborative  research  has  occurred  between  RAND  researchers  in 
the  Santa  Monica  and  Washington  offices  for  years.  Most  of  the  past 
collaborative  efforts  have  involved  different  tasks  performed  at  each  site  with 
the  results  of  the  multiple  tasks  integrated  at  one  of  the  sites.  E-based  research 
capabilities  is  an  alternative  approach  to  perform  research  from  diverse 
locations,  and,  in  some  cases,  use  of  these  techniques  might  be  more  cost- 
effective  and  enable  more  researchers  to  participate.  In  addition,  a  research  task 
might  be  accomplished  in  less  time  than  with  other  methods. 
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E-Based  Research  Requirements 
on  Researchers 

•  Determine  which  functions  are  required 

-  For  example,  instantaneous  file  transfers, 
multiway  voice  communications,  file  sharing, 
video,  secure  chat 

•  Prioritize  functions 

-  Not  everything  can  be  supported 

•Be  adaptive 

-  Learn  new  programs,  procedures,  equipment 
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Researchers  must  also  be  adaptive.  In  providing  e-based  collaborative  research 
capabilities,  management  cannot  be  expected  to  provide  every  option  at  the 
start.  Hence,  researchers  must  prioritize  and  determine  which  functions  are 
most  important.  For  example,  among  instantaneous  file  transfers,  multiway 
voice  communications,  file  sharing,  video  capability,  or  secure  chat,  which  are 
more  valuable  for  conducting  collaborative  research  with  colleagues  in  different 
locations?  In  addition,  to  gain  new  e-based  research  capabilities,  researchers 
may  be  required  to  learn  new  programs  or  procedures  and  familiarize 
themselves  with  new  equipment. 
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E-Based  Research  Requirements 
on  Computer/Network  Support 

•  May  impose  additional,  more  stringent,  or  different 
requirements  on  computer/network  support 

-  Now  focus  on  individual  packages  working  on  individual 
machines 

-  Need  to  add  focus  of  compatibility  among  packages  and  working 
among  many  geographically  separate  machines 

•  In  addition  to  supporting  certain  packages,  might  now  have 
to  support  certain  collaborative  functions 

-  For  example,  only  Netscape  can  be  used  for  file  transfers, 
NetMeeting  can  only  be  used  for  four  or  fewer  parties 

-  Having  more  than  one  party  involved  heightens  importance  of 
quality  control,  thoroughness  in  testing,  and  accuracy  of  self  help 

C.  Wong,  E-Explosion  DB,  03/08/2001  1:16  PM,  63 


Achieving  e-based  collaborative  research  capabilities  might  also  place 
requirements  on  computer  and  network  support  staffs.  One  current  focus  of 
computer  support  is  individual  packages  working  on  individual  machines. 
Another  focus  is  compatibility  among  packages.  Achieving  e-based 
collaborative  research  capabilities  might  heighten  the  visibility  of  compatibility 
among  packages  as  well  as  heighten  concern  about  compatibility  of  subsets  of 
computers  in  many  locations.  In  addition  to  supporting  individual  packages, 
collaborative  functions  might  have  to  be  supported.  For  example,  NetMeeting 
might  have  to  be  supported  for  a  certain  number  of  researchers.  Since  e-based 
collaborative  research  tools  involve  more  than  one  party,  there  might  be  some 
heightened  importance  or  priority  in  quality  control,  thoroughness  of  testing, 
and  accuracy  of  self-help. 
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Summary  and  Concluding 
Remarks 

•  E-based  capabilities  have  the  potential  for 
more  efficiently  and  effectively 
accomplishing  research 

•  More  study  is  needed  to  determine  which 
capabilities  to  pursue 

•  A  strategic  plan  will  help  make  e-based 
research  from  diverse  locations  a  reality 
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E-based  research  capabilities  have  the  potential  to  allow  researchers  to  more 
effectively  and  more  efficiently  accomplish  their  research  tasks.  In  particular, 
e-capabilities  can  ease  accomplishment  of  research  tasks  at  less  cost  and  time. 
Such  e-tools  can  also  allow  more  researchers  to  participate  in  a  research  task. 
More  study  is  required  to  determine  which  e-capabilities  will  provide  the 
greatest  return.  A  strategic  plan  will  help  make  e-based  research  from  diverse 
locations  a  reality. 

Postscript:  A  second  E-DEL+I  exercise  was  conducted  on  March  19,  2001  to 
place  the  Army’s  Applied  Research  Technologies  on  the  same  market  breadth- 
Army  utility  framework.  Fourteen  experts  participated,  some  of  whom  also 
participated  in  the  proof-of-concept  exercise  described  in  this  documented 
briefing.  The  second  exercise  also  generated  data  that  was  used  in  the  Arroyo 
Center  Smart  Outsourcing  Project.  A  representative  from  the  Army  Materiel 
Command,  representing  the  sponsor  of  the  Smart  Outsourcing  Project, 
participated  in  the  second  exercise,  showing  that  participants  without  access  to 
the  RAND  intranet  could  also  successfully  participate.  The  second  exercise 
tested  some  improvements  designed  to  mitigate  the  challenges  identified  in  the 
first  E-DEL+I  exercise.  Phase  I  of  the  “How  Will  the  e-Explosion  Affect  How 
We  Do  Research?”  Project  culminated  with  two  provisional  patents  for  the 
processes  developed. 
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Outline 

•  Introduction 

•  Taxonomy  of  e-Based  Research  Tools 

•  Alternative  Implementation  Strategies 

•  E-DEL+I  Proof-of-Concept  Exercise 

•  Observations 

►Appendix:  Detailed  Description  of 
Research  Framework 
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The  next  nine  figures  constitute  the  appendix  to  this  documented  briefing.  This 
appendix  contains  detailed  descriptions  of  the  seven  steps  of  the  research 
framework  (make  an  observation  and  generate  a  question,  formulate  a 
hypothesis,  make  a  prediction,  test  the  prediction,  analyze  the  test  results, 
interpret  the  analysis  of  the  test  results,  and  draw  conclusions),  as  well  as  the 
“gather  and  analyze  data  and  information”  activity,  which  takes  place  in  each 
step. 
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Generic  Research  Process 

•  Make  observation  and  generate  question 

•  Formulate  hypothesis 

•  Make  prediction 

•  Test  prediction 

•  Analyze  test  results 

•  Interpret  analysis 

•  Draw  conclusions 
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We  examine  the  steps  of  the  generic  research  process  more  closely.  As  the 
research  framework  showed,  the  seven  steps  of  the  generic  research  process  are 
to  make  an  observation  and  generate  a  question,  formulate  a  hypothesis,  make  a 
prediction,  test  the  prediction,  analyze  the  test  results,  interpret  the  analysis  of 
the  test  results,  and  draw  conclusions. 

In  the  following  figures,  we  will  examine  each  of  these  steps. 
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Make  Observation  and  Generate 
Question 

•  Make  an  observation 

•  Generate  question  to  be  answered 

-  Open  ended 


Question 
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Observations  can  be  made  under  any  circumstances.  The  generation  of  a 
question  based  on  an  observation  is  a  thought  process  augmented  by  generally 
brief  data/information  analysis.  Often  the  data/information  considered  is  what 
is  already  known  to  the  observer.  That  is,  at  this  stage,  little  time  might  be 
spent  gathering  data/information  to  generate  a  research  questions.  Research 
questions  usually  evolve  into  open-ended  questions. 

The  product  of  this  step  is  a  research  question  that  is  carried  into  the  next  step, 
hypothesis  formulation. 
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Formulate  Hypothesis 

•  Formulate  tentative  explanation  for  what  is  observed 

-  Gather  information/data 

-  Analyze  information/data 

•  Devise  method  to  test  tentative  explanation 

-  Explore  applicability  of  methodologies 

-  Determine  what  data/information  is  available 

•  Find  existing  knowledge  in  repositories 

-  Print  material,  electronically  based  material,  people 

•  Determine  relevance,  usefulness,  access 

-  Determine  what  new  knowledge  is  needed 

•  Brainstorm 

•  Design  experiment 

Hypothesis 
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Once  a  research  question  is  generated,  the  research  moves  on  to  the  “formulate 
hypothesis”  step.  A  hypothesis  is  a  tentative  explanation  for  what  is  observed. 
To  formulate  a  hypothesis,  a  researcher  might  gather  and  analyze  information. 
This  activity  can  be  extensive  or  minor  in  nature. 

The  task  of  hypothesis  formulation  also  involves  consideration  of  how  the 
hypothesis  might  be  tested.  Part  of  the  task  is  to  devise  candidate  methods  to 
test  the  hypothesis.  This  activity  would  involve  exploring  applicability  of 
established  methodologies  or  deriving  variants. 

The  product  of  this  step  is  a  hypothesis  that  is  carried  into  the  next  step,  making 
a  prediction. 
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Make  Prediction 


•Use  deductive  reasoning  to  predict  outcome 
of  hypothesis  testing 

-  From  generally  known  principles,  extrapolate 
expected  outcome  of  testing 

•  For  example,  if  general  principle  is  true,  then 
specific  hypothesis  test  should  produce  certain  result 


Prediction 
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Once  the  researcher  has  formulated  a  hypothesis  and  established  how  the 
hypothesis  might  be  tested,  he  must  make  a  prediction  about  the  outcome  of  the 
hypothesis  testing.  Generally,  the  researcher  will  use  deductive  reasoning  to 
determine  the  prediction.  That  is,  the  researcher  will  apply  generally  known 
principles  and  extrapolate  an  expected  outcome  of  the  hypothesis  testing.  For 
example,  if  a  general  principle  is  true,  then  specific  hypothesis  testing  should 
produce  a  certain  result. 

The  product  of  this  step  is  a  prediction  that  is  carried  into  the  next  step, 
performing  the  hypothesis  testing. 
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Perform  Test 

•  Determine  methodology 

•  Assemble  information,  equipment,  materials 

•  Perform  test 

-  For  example,  perform  experiment,  exercise 


Test  results 
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The  initial  task  in  the  “perform  test”  step  is  to  decide  on  a  methodology. 
Normally,  the  methodology  selected  will  be  one  of  the  candidate  techniques 
identified  in  the  hypothesis  formulation  step.  Minor  modifications  may  be 
required  for  practicality,  cost,  schedule,  or  other  resource  considerations. 
Researchers  must  be  attentive  that  the  modifications  do  not  affect  the 
applicability  of  the  methodology. 

Sometimes,  the  test  is  performed  by  the  researchers  themselves,  and  other  times 
it  may  be  performed  by  a  group  designated  and  monitored  by  the  researchers. 

In  either  case,  the  required  information,  data,  equipment,  materials,  and  supplies 
must  be  assembled  and  the  participants  must  be  identified  and  available. 

Finally,  the  test  is  performed. 

The  product  of  this  step  is  the  raw  test  results  that  are  carried  into  the  next  step, 
“analyze  test  results.” 
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Analyze  Test  Results 

•  Organize  and  structure  results 

•  Combine  results  with  other  data/information 

-  Make  logical  deductions 

•  State  universal  facts,  deduce  the  specifics  they  imply 
based  on  set  of  assumptions 

-  For  example,  theoretical  proofs 

-  Create  inductive  arguments 

•  State  cases  and  generalize  to  universal  conclusion 

Analysis  of  test  results 

C.  Wong,  E-Explosion  DB,  03/08/2001  1:16  PM,  71 


The  raw  test  results  need  to  be  organized  and  structured.  Analysis  methods 
applicable  to  this  step  include  statistics  generation,  sorting  methods,  and 
classification  techniques.  The  test  results  might  be  combined  with  other  data  or 
information  to  make  logical  deductions.  For  example,  the  researcher  might 
consider  universal  facts  and  deduce  the  specifics  they  imply  based  on  a  set  of 
assumptions.  Inductive  arguments,  in  which  the  researcher  states  specific  cases 
and  generalizes  them  to  universal  conclusions,  may  also  be  employed. 

The  product  of  this  step  is  an  analysis  of  the  test  results,  which  is  input  to  the 
“interpret  analysis”  step. 
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Interpret  Analysis 

•  Inference 

-  State  reasoning  from  analytic  evidence  to  conclusion 

-  Integrate  analysis  with  particulars  of  situations  being 
considered  and  questions  being  answered 

•  Extend  analysis  with  respect  to  a  particular 
dimension 

-  Determine  robustness  of  analysis 

•  For  example,  extrapolate  trends,  vary  assumptions,  perform 
sensitivity  studies 

Interpretation  of  analysis 
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When  interpreting  the  analysis  of  the  test  results,  the  researcher  establishes  a 
chain  of  reasoning  from  the  analytic  evidence  to  possible  conclusions.  The 
researcher  might  integrate  the  analysis  with  particulars  of  the  situations 
considered  and  the  question  being  addressed.  To  help  with  the  interpretation, 
the  researcher  might  attempt  to  extend  the  analysis  with  respect  to  particular 
dimensions.  The  researcher  might  attempt  to  determine  the  robustness  of  the 
analysis  by  extrapolating  trends,  varying  the  assumptions,  and  performing 
sensitivity  analysis.  These  techniques  will  help  the  researcher  establish  the 
extent  of  the  validity  of  the  analysis. 

The  product  of  this  step  is  an  interpretation  of  the  analysis  that  is  carried  onto 
the  final  step  of  the  research  process,  “drawing  conclusions.” 
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Draw  Conclusions 

•  State  what  was  proven 

•  State  implications  of  proven  facts 

-  For  example,  policy  implications 

•  State  usefulness  of  methodology 

-  For  example,  scope  of  applicability 

•  State  next  steps  and  recommended  actions 

-  For  example,  what  needs  to  be  proven  next,  what 
questions  need  to  be  considered  next 

Conclusions 
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At  this  step,  the  researcher  states  what  has  been  proven.  The  researcher’s 
underlying  assumptions  must  be  made  clear  to  effect  an  unambiguous 
understanding  of  the  proven  facts.  The  researcher  may  also  state  the 
implications  of  what  has  been  proven.  For  example,  policy  implications  might 
be  discussed.  The  usefulness  of  the  test  methodology  employed  and  the  scope 
of  its  applicability  may  also  be  part  of  the  researcher’s  conclusions.  Finally,  the 
researcher  may  state  the  next  steps  or  recommend  actions  about  what  needs  to 
be  proven  next  or  what  questions  should  be  considered  in  the  future. 

The  product  of  this  step  is  the  researcher’s  set  of  conclusions. 
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Gather  and  Analyze  Data  and 
Information 

•  Existing  knowledge 

-  Find  information  or  data  in  a  repository 

•  Print  material,  electronically  based  material,  people 

-  Determine  relevance,  usefulness,  access 

•  New  knowledge 

-  Generate  new  information  and  data 

•  Brainstorm 

•  Design  and  perform  experiments 


Data  and  information 
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Throughout  the  research  process,  the  researcher  is  gathering  and  analyzing  data 
and  information.  The  research  must  consider  what  information  or  data  is 
available  for  each  step  of  the  research  process  and  how  much  time  and  effort 
will  be  afforded  to  evaluate  that  information.  Some  of  this  information  might 
exist  in  repositories  such  as  print  libraries,  or,  electronically,  in  databases. 
People  can  also  furnish  information,  and  more  involved  data  collection  might 
be  required  to  gather  information  from  a  human  source. 

The  relevance,  usefulness,  accuracy,  and  access  to  the  information  must  also  be 
established. 

Finally,  new  knowledge  might  be  required  to  perform  a  step  adequately.  New 
knowledge  might  stem  from  brainstorming  activities  or  from  experiments. 

The  product  of  this  activity  is  data  and  information  that  will  be  used  to  perform 
each  step  of  the  research  process. 


74 


